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Abstract. /n physics and chemistry, the 
development of problem-solving skills is 
necessary to become an expert. A simple 
cognitive model to analyse such develop- 
ment is proposed and tested. An explora- 
tory research was conducted with expert 
professors and students in initial and 
advanced years. A think aloud procedure 
was used to obtain relevant data while the 
participants tried to solve undefined, open 
problems. Solving these problems required 
a particular skill representative of exper- 
tise: modelling reality using science. More 
than 1350 solving actions were collected 
and related to the mental representations 
elaborated, developed and inter-related by 
solvers. The proposed model was able to ac- 
count for expert-novice differences in terms 
of the respective distributions of solving 
actions among the mental representations. 
Large differences appeared in the mental 
representation of Conceptual scientific 
Model. In addition, advanced and initial 
students showed similar and significant 
averages of unproductive actions, while 
experts took very few. Experts showed 

high convergence in their distributions of 
actions among the mental representa- 
tions. If the outcomes were replicated with 
higher external validity, the model could 
help researchers to analyse the cognitive 
mechanisms in problem-solving, and 
teachers to better focus their efforts on 
specific students’ lacks. 
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Introduction 


In physics and chemistry, the development of problem-solving skills is 
necessary to become an expert. Experts can be distinguished from less expe- 
rienced individuals for their competent behaviours in problematic situations, 
which are based on the amount of accumulated domain-specific knowledge 
and how this knowledge is used in these situations (Ericsson et al., 2018). 
Experts differ from novices not only in the amount of accumulated (correct) 
knowledge, but also in the way that knowledge is structured in their minds 
(Bogard et al., 2013). Experts create larger and more interconnected mental 
representations than novices, which have deeper and more detailed hierarchy 
levels and are easily oriented to action (Bjorklund, 2013), in form of schemata. 

This organized knowledge can be used at once when activated by 
singular ‘triggering elements in a problematic situation. Thus, experts tend 
to show a schema-driven behaviour when dealing with new problems, as 
pointed out by Larkin et al. (1980) in an early research with physicists. On the 
other hand, novices tend to be more case-driven; in other words, they tend 
to be influenced by surface similarities of some prior and familiar cases (Ball 
et al., 2004). Therefore, novices show more dispersion than experts in the 
type of mental or physical actions taken to solve a problem. Mental actions 
of experts tend to converge in the analysis to tackle problems, while mental 
actions of novices are more divergent (Hong & Liu, 2003), which is probably 
due to the novices’ trial-and-error procedures. Experts’ convergent thinking 
makes possible students’ guidance toward a defined goal of expertise in 
particular tasks. 


Problem-solving by Experts and Novices in Physics and Chemistry 


This research addresses the differences between experts and novices 
in problem-solving behaviour in the subjects of physics and chemistry. 
Problem-solving is a usual task used often in academic contexts for teach- 
ing and evaluating learning. It is also the main task in carrying out scientific 
research. Solving a problem implies the combination of several skills usually 
attributed to experts, such as focusing on key features to classify a problem 
as a situation that involves a specific well-known phenomenon (coding), or 
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using schematized scientific and strategic knowledge to take the actions needed in order to find a solution. Thus, 
solving ‘difficult, non-straightforward problems is an indicator of acquired expertise. 

Chi et al. (1982) have summarized important differences between experts and novices in physics problem- 
solving. Despite the obvious differences in the amount of scientific knowledge, two other differences have been 
found: 

1) Coding and problem representation: Experts and novices create different mental representations of the 
same problem (Chi et al., 1981). Knowledge organization in the minds of experts implies the perception of certain 
structural problem elements bearing categorical relevance (Boshuizen et al., 2006). Experts categorize problematic 
situations according to basic science principles and laws, which determine specific relations among magnitudes 
and quantities, while novices tend to focus on surface explicit features (e.g. slopes and springs). Thus, it is difficult 
for novices to take advantage of previously studied worked-out analogues and use them as source-examples in 
analogical transfer (Gomez-Ferragud et al., 2013). Coding differences imply differences in the mental representa- 
tion of a new problem, different knowledge forms activated, thus leading to different solving actions that result 
in success or failure in the task (Gomez-Ferragud et al., 2014). 

Il) Solving strategies: Novices frequently show a lack of strategic knowledge, in addition to little science knowl- 
edge. When compared to experts, novices have difficulties in selecting and applying useful strategies, and making 
key inferences necessary to progress in the task. In physics, Larkin et al. (1980) found that experts usually perform 
a qualitative analysis of a new problematic situation in terms of some scientific conceptual model before dealing 
with mathematical equations. Similar findings were gained by Randles and Overton (2015) in chemistry. Among 
experts, the scientific model was coherently used to understand the problem statement, perform a qualitative 
analysis, and formalize the scientific model into equations to be solved. 

In contrast, novices frequently carry out the process of searching-and-selecting for an equation in which the 
givens of a problem fit. Novices mix their (alternative) ideas based on the daily-life world with ideas from phys- 
ics instruction so creating incoherent models that usually cause a lack of deep understanding. Thus, they rely on 
mathematics (Champagne et al., 1983; Randles & Overton, 2015). In chemistry, similar differences were observed. 
While experts address new problems using conceptual knowledge and qualitative analysis in order to achieve the 
solution, novices use less successful mean-ends procedures with vague and partially wrong qualitative analyses 
(Heyworth, 1999). The initial qualitative analysis implies that experts take relatively longer than novices to re-define 
the problem in terms of the scientific knowledge (Brand-Gruwel et al., 2005). 

Finally, the schema-driven solving procedure permits experts to avoid their working memory being overloaded 
by irrelevant data (Randles & Overton, 2015). Therefore, they can focus their efforts on crucial points and manage 
more cognitive resources than novices via monitoring and self-regulation (Brand-Gruwel et al., 2005). 


Research Focus 


Previous studies described distinctive behaviours between experts and novices in solving physics or chemistry 
problems. However, unveiling the underlying mental processes that cause these differences is more challenging. 
The present exploratory research aimed at testing a simple cognitive model proposed to analyse the develop- 
ment of problem-solving skills in university students’ way towards expertise. The model is based on the idea that 
solving actions highlight important content of the mental representations the solver has to build, develop and 
relate in order to understand and solve a problem. Therefore, each solving action can be related to some mental 
representation. Solving actions have been studied and classified into different taxonomies. On the other hand, 
the mental representations needed to understand and solve physics problems have been also described in differ- 
ent cognitive models, as the one by Greeno (1989) or the model proposed by Truyol, et al. (2014). In this way, the 
proposed model implies a step beyond with respect to previous research based on solving actions only (Meijer et 
al, 2006; Randles & Overton, 2015), and with respect to other studies based solely upon the mental representations 
elaborated when performing academic tasks (Ibrahim & Rebello, 2013). 

In the university context, expertise is frequently assessed by the ability to solve difficult problems in new 
contexts. Hence, the focus in the present research is to test the ability of the proposed model to capture possible 
differences in the content of the mental representations that expert and novice solvers have to build, develop 
and inter-connect in order to understand and solve physics or chemistry problems at the university. If the model 
accounted for the expected expert-novice differences, then it would shed some light on the origin of previously 
documented students’ ineffective problem-solving behaviours. The long-term processes for expertise acquisition 
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within the university setting could be accelerated by professors, as expert performers, if they were able to realize 
where students’ lack of knowledge and inefficient use of skills are localized when solving academic problems. 

Undefined physics and chemistry problems were used in the present research because they are more demand- 
ing than defined ones (which sometimes seem to be mere exercises) and they present a potential to maximize 
experts-novices differences, especially in the skill of modelling reality with science. 


The Model 


As stated previously, the proposed model is based on the relation between solving actions and mental rep- 
resentations. The following lines elaborate on these two components. 


Mental Representations 


Solving a problem implies understanding the situation posed and the question asked, planning the way from 
the actual initial state to the final state, navigating in the problem space towards the goal, and reflecting on the 
coherence and correctness of the solution achieved (Polya, 1945). Understanding provided information implies the 
elaboration of mental representations capable of integrating the solver’s prior-to-new knowledge in a coherent way 
(Kintsch, 1998). In the case of problems, understanding information given in the statement and navigating in the 
problem space implies being able to develop and coherently link several mental representations (Greeno, 1989). In 
the present research, we assume the model proposed by Truyol, et al. (2012; 2014) for problem-solving in physics. 

This model considers that understanding and solving a problem implies the construction of three levels of 
mental representations: Situation Model (SM), Conceptual Model (CM), and Formalized Model (FM). Furthermore, 
it implies the coherent link and transitions between the three models (see also the model proposed by Greeno, 
1989). There are forward nature transitions: Building CM from SM (BCM) and Building the FM from CM (BFM). There 
are also transitions of backward nature: Interpreting the formal outcomes in terms of science laws (IN) and con- 
necting scientific results to particular real-life situations, or Instantiation (IS). 

In the present research this model has been assumed as valid for chemistry problems as well, although chem- 
istry and physics have been recognized as having epistemological differences (Jensen, 1998; Taber, 2013). Assuming 
that problematic situations can be conceptualized and represented in physics or in chemistry using elements of 
different ontological and epistemological nature, it seems that problem-solving in chemistry requires, as in physics, 
modelling reality using chemical principles, laws, and concepts, as well as formalizing conceptual chemical models 
and linking outcomes with reality in order to check the proposed solutions. 

Table 1 describes the mental representation components and their characteristics as postulated in the model. 
It also includes some specific examples. 


Table 1 
Mental representations components and characteristics in the model of Truyol et al. (2014) 


Characteristic 


Situation Model 


Conceptual Model 


Formalized Model 





Components Real-life: people and their Scientific models of real-life objects, events ~ Logical and mathematical representation of 
actions, objects and their and their features and characteristics. scientific entities: quantities, relationship among 
attributes, and events and their Examples: iron — Fe; oxidation-reduction, | magnitudes, laws for events expressed as equa- 
characteristics. standard reduction potential; burning tions or geometric relations, etc. 

Examples: iron (hard; rusted), combustion: mixing with O, and energy Examples: reduction potential Fe — Fe**(aq) 

butane (gas; burning), car obtained; Car moving — point-like particle  +2e — Fe(s); E,= -0.41 volts; butane combustion 

(moving fast on a road). Galilean kinematics, velocity. — 2C,H,, + 130, <> 8CO, + 10H,O; point-like 
particle kinematics > v= v, + (1/2)a ? 

Guided by Everyday rules on how the Principles and scientific laws expressed in Mathematical rules and formal Logics. 


world is and works. Common 
sense, logics. 

Examples: The screw is red, it 
has rusted; With the stove fire 
you can cook; Using the car 
you will arrive sooner. 


conceptual terms. Conditions of application 
or validity. Conditions of applicability of 
scientific models. 

Examples: Electrochemistry; Law of con- 
servation of mass; Galilean Kinematics and 
Newtonian mechanics. 


Examples: AG=nFE,.,; 
>mass reactants=) mass products; t? = 2(v-v,)/a 
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Characteristic Situation Model Conceptual Model Formalized Model 
Ontological Concrete category (non- Concrete and abstract categories. Only abstract categories 
categories abstract): components percep- 
tible through senses. 
Language Natural language, figurative Technical-scientific language. Logical and mathematical symbolic language. 
images and pictures. Graphs, tables, equations, geometry. 
Utility Informal estimations; qualita- Estimations of orders of magnitude in Quantitative, more precise descriptions, explana- 


tive descriptions, analyses and _ attributes and characteristics of scientific tions and predictions in scientific phenomena. 
prediction in the real-life world. | phenomena. 


Source: elaborated from Truyol, et al. (2014; p. 884), Table 1. 


Actions Taken when Solving a Problem 


The use of specific solvers’ skills can be observed by means of the actions taken in the process of solving 
particular problems. Research on the solving processes has been often addressed using think-aloud technique: 
the solver voices what they are thinking to experimenters and justifies the actions taken. Therefore, the greater 
ability to think about one’s thinking, the richer data provided to experimenters. When using the think-aloud 
technique, the solvers’ perceived skills are mixed with their metacognitive abilities (Desoete, 2008; Schellings, 
et al., 2013). In other words, the observed actions taken are triggered by the solver’s metacognitive activity. 

In the present research, Meijer, et al’s (2006) taxonomy for the actions taken while solving a problem was 
used. This taxonomy was chosen for three reasons: a) Its elaboration and validation processes, specifically for the 
analysis of think-aloud protocols in problem-solving and its high reliability; b) Its metacognitive and cognitive 
foundation, as the observable data obtained in think aloud procedures are triggered by the subject's metacog- 
nitive activity; and c) Its hierarchical structure, which allows for a macro-analysis based on over-arching, main 
categories, and also for a micro-analysis based on embedded minor categories when necessary. Actions are 
classified in 6 main categories: orientating (OR), planning (PL), executing (EX), monitoring (MO), evaluation (EV) 
and elaboration (EL). Each of these main categories includes specific subordinate categories (see the Appendix 
in Meijer et al., 2006). 

Other taxonomies for the solving activity have been proposed in the literature (Ali et al., 2018; Rosenzweig 
et al., 2011). In a think-aloud research developed with undergraduate students, industrialists, and academic 
chemists, Randles and Overton (2015) elaborated a taxonomy of positive (made/useful) and negative (absent/ 
useless) actions of experts in solving open-ended problems. Although this taxonomy was based on pragmatic 
foundations (rather than purely cognitive), it was useful for the research purposes and specific differences in 
solving actions taken by each of the groups, were found. According to the study, academics, acting as experts, 
focused on identifying the information needed, planning how to tackle the problem, identifying and framing 
the problem, making approximations and estimations, evaluating the strategy as well as the solution achieved, 
applying a logical scientific approach to their solution, and not becoming distracted by the lack of details in 
the question. In turn, the behaviour of undergraduates mainly involved identifying information needed, using 
equations and calculations, being confused on how to tackle the problem, identifying and framing the problem, 
using approximations and estimations, and developing a useful strategy. However, drastic differences were not 
found in what they do for solving the problems. Nonetheless, academics showed subtle differences in the time 
and effort devoted to certain specific actions that students took less frequently. 


Defined and Undefined Problems in Physics and in Chemistry 


Johnstone (1993) has provided a useful classification of problems in eight different types (2x2x2) accord- 
ing to the data given (complete/incomplete), the methods to achieve the solution (familiar/unfamiliar to the 
solver) and the unknowns (clearly defined/undefined, and open). In this classification, most problems used in 
university-level physics and chemistry instruction can be classified as algorithmic exercises; these are: ‘well- 
structured; ‘closed’ or‘single-possibility’ problems (Shekoyan & Etkina, 2007). In physics and chemistry, a defined 
(or well-defined/well-structured) problem has a statement that defines a situation in terms of an explicit scientific 
model (e.g. redox or Galilean kinematics) and poses specific questions (the unknowns) in terms of numerical 
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values for some magnitude coherent with the model. The necessary data to achieve the solution is provided 
and also fits the scientific model. Solving a defined problem usually implies the skill of building and developing 
the formalized model from the given scientific model. It also involves the interpretation of the mathematical 
outcomes in the light of the scientific model. 

More interesting for students’ acquisition of expert skills are problems those statements neither suggest 
the scientific model, nor provide the givens necessary to reach a solution. These are‘undefined’ ‘open-ended; or 
‘multiple possibilities’ problems (Ringenberg & van Lehn, 2008; Shekoyan, 2009), and have been pointed up as 
the only problems able to develop scientist skills (Becerra-Labra et al., 2012; Gil-Pérez & Torregrosa, 1983). In an 
undefined problem, the problematic situation is defined in terms of the real-world. Then, solving the problem 
requires the skill of modelling reality by using science. This involves several actions as creating and developing 
a scientific conceptual model, building the formalized model from the conceptual model, and navigating this 
formalized model. Navigating the problem space usually means performing qualitative reasoning to link mag- 
nitudes in causal chains that obey scientific laws and principles and elaborating estimations with magnitude 
orders. For experts, solving an undefined problem also entails making assumptions to constrain the degrees 
of freedom in order to convert the ill-defined, open-ended, multiple-possibilities problem into a well-defined 
one (Fortus, 2009). These actions are associated with building and developing a conceptual scientific model. 
The final step is an instantiation process in which the expert establishes a backward connection of the science 
outcomes with the reality described in the statement of the problem (Truyol, et al., 2012). 

In an empirical exploratory research Truyol (2012) found that university students showed significant 
differences in their success solving defined or undefined physics problems. Looking for the causes of these 
differences she first analysed the worked-out examples and the problems posed to the students along the 
university degree. The author concluded that undefined problems were clearly infrequent compared to the 
defined ones. Second, Truyol et al. (2014) investigated the possible different skills needed to solve defined 
or undefined problems. They conducted a case study with several expert university professors. The analy- 
ses used the aforementioned model (Table 1). Results showed differences between defined and undefined 
problem-solving regarding the distribution of solving effort in the mental representations and transitions. 
However, the differences did not appear in building or navigating the FM, or when interpreting the numeri- 
cal outcomes. Hence the authors concluded that the skill “modelling reality with physics” could be poorly 
developed even at university. 


Research Questions 


The present research sought to answer the following research questions: 
Q1: Can the proposed model classify all the data provided by the informants? 
Q2: Howare the solvers’solving actions distributed among mental representations when undefined physics 
or chemistry problems are solved by experts and novices with different levels of knowledge? 
Q3: Does the model account for problem-solving differences between experts and novices with different 
levels of knowledge? 


Research Methodology 
General Background 

The present research was exploratory in nature and based on the analysis of the information provided by a 
group of problem-solvers in individual interviews conducted by two of the researchers. Qualitative (coding) as well 
as quantitative techniques (proportions; non-parametric statistics) were used. 

Participants 

Eleven experts, eleven 2™ year university students (initial students onwards) and 7 advanced university 

students in the final year (5) of their respective degrees participated in this exploratory research. Although this 


is a small sample in terms of participants, they provided a high number of information units required to test 
the proposed model in a first approach. Experts were physicists or chemists, and university teachers with more 
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than 5 years of professional experience. Students were enrolled in Physics or Chemistry university degrees. The 
initial students had studied the topics implied in the experimental problems already. 

The solving sessions took place outside the classroom time. Before starting, all of them were individually 
informed of the only academic purpose of the research and invited to participation as anonymous volunteers. 
Each student was informed of the total absence of academic consequences of his/her performance in the 
problem-solving session. When a participant asked for the outcomes, he/she was informed of the global results, 
preserving the anonymity of the participants. Although this small sample of participants lacks sufficient external 
validity, it can provide data suggesting further research. 


Instrument and Procedures 


Two physics and two chemistry undefined problems were defined following the procedure described in 
Truyol et al. (2014). The Appendix shows examples of the problems used. For each participant, the relevant data 
were: their level of expertise (low for initial students, intermediate for advanced students, and high for expert 
professors); the types of actions taken (6 superordinate categories defined in Meijer et al’s taxonomy), useful- 
ness of the actions (productive or unproductive in order to solve the problem) and the mental representation 
or transition associated to the action taken according to the model described in Truyol et al. (2014). An action 
was considered as “productive” when it led the solver to a better understanding of the problematic situation, or 
when it placed the solver closer to the problem goal (i.e. the action was profitable). Otherwise, the action was 
classified as “unproductive”. Unproductive actions included incorrect actions. 

Data collection was carried out along two years, according to the participants’ availability. The interviewers 
made individual appointments with the informants respecting their work and classroom schedules. Individual 
interviews were conducted in a quiet room. A video camera was placed in a zenithal position to record the par- 
ticipants’ solving actions on a paper and their speech, but not their faces to preserve anonymity. Each participant 
solved only one problem in their academic field (physics or chemistry). It was considered enough in the present 
exploratory research, as more than 1350 information units were collected and analysed in three different ways 
(i.e. more than 4000 classifications) to test the proposed model. Participants were as far and deep in the solving 
task as they could, and the process was time-consuming: 30-40 min per session. 


Data Analysis 


Protocols were segmented into units, each unit corresponding to a single action. Thus, a particular category 
of action was assigned to each unit (see previous subsection): ORientating, PLanning, EXecuting, MOnitoring, 
EValuation or ELaboration. Next, the action was associated to a specific mental representation or transition 
between them (see previous subsection): Situation Model, Building the Conceptual Model, Conceptual Model, 
Building the Formalized Model, Formalized Model, Interpretation or InStantiation. Finally, the action was con- 
sidered as productive or unproductive to solve the problem. Therefore, each action was classified three times, 
according to their typology, then implied mental representation and its usefulness. 

Two raters independently classified the units and the corresponding Cohen's kappas were computed for 
a 30 percent of the units selected at random. Disagreements initiated a discussion and improvement of the 
classification rules. Next, a second 30 percent of the units were classified. The final kappa values obtained 
were .78 for types of action (p< .001; substantial agreement) and .85 for mental representations or transitions 
(p< .001; almost perfect agreement) and the same value was obtained for the usefulness of the actions taken. 

Examples of codified information units are given in the Appendix. 

As the participants’ sample was small in the present research, we used non-parametric statistics in order 
to support some observed trends in the data (Mann-Whitney and Wilcoxon tests). 


Research Results 
As the solutions achieved concerns, the levels of success were diverse. Only one initial student was able 


to deal with the experimental problem and achieved a correct, but incomplete, solution. The remaining initial 
students abandoned the solving procedure at an intermediate point. The advanced students attained partial 
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solutions to the problem, with only one exception who gave up after feeling stuck. Experts found a complete 
and correct solution for their respective problems, as expected. 


Type of Actions Taken 


One initial student was considered as an outlier as their total amount of actions exceeded 3 standard devia- 
tions from the global mean value. This participant was excluded from the analyses. 

Experts took a higher total amount of actions (n= 634) with a higher average (M= 57.6, SD= 17.8) than 
Advanced students (n= 370; M=52.9; SD= 26.9) or Initial students (n= 361; M= 36.1; SD= 18.0). 

Experts showed significantly higher averages of productive (M= 52.6) than of unproductive actions (M= 
5.0; Wilcoxon test: Z= -2.934; p= .003), and the same happened for advanced students (37.0 and 15.9; Z= -2.371; 
p= .018). However, in initial students the averages of productive (M= 19.2) and unproductive actions (M= 16.9) 
were statistically similar (Z= -1.425; p= .154). Initial and advanced students significantly differed in the average 
of productive actions taken (Mann-Whitney: U= 9.0; p= .011) but showed comparable averages of unproduc- 
tive actions (U= 32.0; p= .769). Experts showed significantly higher average of productive actions than initial 
(Mann-Whitney; U= 4.0; p< .001) or advanced students (U= 16.5; p= .044), and significantly lower average of 
unproductive actions (U= 6.5; p< .001; U= 2.5; p< .001). In terms of percentages, 91.2% of the actions taken by 
experts were productive, compared to 53.2% and 69.9% for initial and advanced students, respectively. Experts 
took few unproductive actions and the rare errors made were immediately corrected because of their effective 
monitoring. 

According to the types of actions considered, the averages are shown in Table 2. 


Table 2 
Average of actions taken per participant according to the level of expertise 


OR PL EX MO EV EL 
2™ year-Students 5.4 74 12.2 5.6 2.6 2.9 
Advanced-Students 9.3 12.1 14.3 8.9 4.3 40 
Experts “7 8.6 20.9 5.2 6.0 9.2 


Comparisons for the amounts of each type of action taken were performed. Initial and advanced students 
showed no significant differences (Mann-Whitney: U> 17.0; p> .075 in any type of action). Experts obtained a 
significantly higher average of Elaboration actions than advanced students (U= 0.5; p< .001) and also higher 
averages of Executing, Evaluation and Elaboration actions than initial students (U< 22.0; p< .020 in these types 
of actions). 


Mental Representations Associated with the Actions Taken 


Actions were associated to one mental representation (SM, CM, FM) or to one transition between mental 
representations (BCM, BFM, IN, IS). Some actions (personal comments, feelings, self-evaluations of the own abili- 
ties, anecdotal comments, etc.) could not be associated to any mental representation concerning the experimental 
problem and were classified as “no mental representation implied” (NM). Table 3 shows in some detail the mean 
values of the actions taken by experts, advanced and initial students in each of the mental representations and 
transitions (except for the NM category), when they try to solve the undefined problem. Table 3 also shows those 
comparisons achieving significance. 
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Table 3 


Mean amount of actions implied in each mental representation or transition, according to the level of expertise. Pair compari- 
sons with significant differences are shown 


Average per participant 
Actions taken in each mental representation and transition 
Initial Advanced Experts 
Elaborating a useful Situation Model (SM) TOT 7.20 10.29 1.64 
4.30 7.86 1.64 
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UNP: Experts << Initial = Advanced UNP 2.90 2.43 0.00 
Recognition of objects and events as part of the real world; understanding; Making sense; Giving PRO 2.50 5.14 1.36 
meaning to the ideas in the statement (OR); 
Looking for particular information in the statement; Self-questioning (PL) UNP 0.50 1.14 0.00 

PRO 1.30 1.00 0.27 
(Self)-Explanations on the problematic situation or the demand (EX) 

UNP 1.20 0.43 0.00 

PRO 0.10 0.29 0.00 
Claiming (partial) understanding; Commenting on task demands (MO) 

UNP 0.40 0.14 0.00 
Finding similarities with other real situations (EV); PRO 0.40 1.43 0.00 
Elaborations and inferences from ideas in the statement (EL) UNP 0.80 0.71 0.00 
Building the Conceptual (scientific) Model (BCM) TOT 10.00 10.00 6.18 
PRO: Initial = Advanced = Experts PRO 7.00 7.86 9.45 
UNP: Experts << Initial UNP 3.00 2.14 0.73 
Identifying an underlying phenomenon; Determining characteristic properties and magnitudes of PRO 4.10 5.00 2.00 
objects and events; Hypothesizing (OR) ‘ ; ‘ 
Determining a specific magnitude as associated with the problem demand; Discriminating relevant 
features to be modelled from distracting details; Outlining a solving schema (PL) UNP 0.90 0.14 0.19 
Transferring from one representation to another; Outlining, drawing a diagram; Associating magni- PRO 2.40 271 2.73 
tudes to features (EX) UNP 1.10 0.71 0.00 
Monitoring aloud the process; Noticing lack of knowledge or retrieval failure; Troubles about PRO 0.20 0.14 0.73 
information provided (MO) UNP 0.40 0.14 0.27 
Finding differences and analogies with previous problems (EV) PRO 0.30 0.00 0.00 
Inferring (EL) UNP 0.60 0.14 0.27 
Developing the Conceptual (Scientific) Model (CM) TOT 6.10 8.71 23.00 
PRO: Initial < Advanced << Experts PRO 2.90 6.29 21.64 
UNP: Initial = Advanced = Experts UNP 3.20 2.43 1.36 
Identifying scientific principles and laws; Activating scientific conceptual knowledge (OR) PRO 0.50 3.29 791 
Imposing restrictions; simplifying; Explaining how a scientific model can solve the problem (PL) UNP 1.00 0.86 0.64 
Connecting magnitudes using scientific laws and principles; Science-based reasoning to describe, PRO 2.00 2.43 7.55 
explain and predict events; Performing estimations of the unknown magnitude; Running the model 
qualitatively; Analysing extreme situations; making approximations (EX) UNP 1.30 1.00 0.00 
Checking the model; Making amendments or corrections; Noticing retrieval failure or lack of knowl- PRO 0.10 0.14 2.09 
edge; Inconsistency detection; Comprehension failure (MO) UNP 0.70 0.43 0.45 
Inferring; Evaluating the model when running it (EV) PRO 0.30 0.43 4.09 
Model-based explanations; Connecting the model with (ideas in) the statement (EL) UNP 0.20 0.14 0.27 
Building the Formalized Model (BFM) TOT 3.40 4.29 4.45 
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PRO: Initial < Experts PRO 1.40 3.14 4.45 
UNP: Experts < Initial = Advanced UNP 2.00 1.14 0.00 
Giving values to magnitudes; Hypothesizing (OR) PRO 1.20 1.43 1.18 
Using information sources to find suitable numerical values for the magnitudes when necessary: 
looking for tables or graphics; Choosing units; Giving meaning to graphs, axes, setting up a 
coordinate system (PL) ON uae oe 0.00 
Writing algebraic (equations or inequations) or arithmetic or graphical connections among magni- PRO 0.10 1.71 3.27 
tudes or quantities or chemical reactions in symbolic terms (EX) UNP 0.50 0.43 0.00 
Numerical quantities required but not found; Noticing retrieval failure of formulae associated with a PRO 0.10 0.00 0.00 
scientific law; Giving meaning to symbols or formulae (MO) UNP 0.70 0.57 0.00 

PRO 0.00 0.00 0.00 
Justifying and evaluating the chosen numerical values or the written algebraic relations (EV) 

UNP 0.40 0.00 0.00 
Developing the Formalized Model (FM) TOT 3.30 7.57 8.55 
PRO: Initial = Advanced < Experts PRO 2.00 5.29 8.55 
UNP: Experts < Initial = Advanced UNP 1.30 2.28 0.00 
Activating knowledge on mathematical expression of laws, functional inter-dependence of magni- PRO 0.30 200 0.55 
tudes, geometrical shape of entities, etc. (OR) 
Formulating a plan to formally solve the problem (outlining the formal trans-formations); Sub- 
goaling; Using books, tables, internet to explain formal trans-formations (PL) UNP 0.00 0.43 0.00 
Performing only some algebraic transformations or arithmetic computations; Elaborating tables or PRO 1.50 2.57 6.27 
graphics; Charge and mass balancing in chemical reactions (EX) UNP 0.70 0,57 0.00 
Obtaining an specific algebraic or arithmetic or graphical relationship for the unknown, used to PRO 0.00 0.00 1.00 
answer the problem (EX) UNP 0.10 0.00 0.00 
Monitoring the correctness and usefulness of the formal transformations; Noticing inconsistencies PRO 0.20 0.71 0.73 
or errors (MO) UNP 0.50 1.29 0.00 
Interpretation (IN) TOT 1.40 3.42 3.55 
PRO: Initial << Advanced = Experts PRO 0.30 271 3.55 
UNP: Experts << Initial = Advanced UNP 1.10 0.71 0.00 
Inconsistency or Error detection and correction when trying to make sense out of the quantities and PRO 0.00 0.29 0.18 
relationships (MO) UNP 0.20 0.14 0.00 
Using science to evaluate the obtained or chosen numerical values, or to assess the relationship PRO 0.30 2.43 3.37 
among magnitudes (EV) UNP 0.90 0.57 0.00 
Instantiation (IS) TOT 1.10 2.14 4.82 
PRO: Initial = Advanced << Experts PRO 0.40 1.00 4.82 
UNP: Experts < Initial = Advanced UNP 0.70 1.14 0.00 
Elaborations to explain the meaning of the obtained outputs in the real world: connecting back to PRO 0.40 1.00 4.82 
the objects and events in the statement; Concluding, trying to satisfy the problem demand (EL) UNP 0.70 1.14 0.00 


The notation “<; <<; =" respectively mean p< .05; p< .01; p> .05 according to Mann-Whitney tests. 


Figures 1 and 2 show the data in a visual format for easier comparison. In productive actions implied devel- 
oping the conceptual model (CM), in building and developing the formalized model (BFM and FM), and in back 
transitions (IN and IS) there was a progression from initial to advanced students, and to experts as shown in Figure 
1. Differences between initial and advanced students achieved significance only in SM (U= 11.5; p= .020), CM (U= 
12.0; p= .024) and IN (U= 8.0; p= .005). However, when BFM and FM actions were considered together, advanced 
students showed significantly higher averages of Executive formal actions than initial students (U= 14.0; p= .037). 
In turn, experts-advanced students’ differences in productive actions were significant in SM (U= 3.0; p< .001), CM 
(U= 1.5; p< .001), FM (U= 15.5; p= .036) and IS (U= 5.0; p= .02). 
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Figure 1 
Mean amounts of productive actions taken by Experts, Initialand Advanced students according to type and mental represen- 


tation and transition involved 
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Figure 2 
Mean amounts of unproductive actions taken by Experts, Initialand Advanced students according to type and mental repre- 


sentation and transition involved 
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In unproductive actions, the opposite progression was obtained, as depicted in Figure 2. However, differences 
between initial and advanced students did not reach significance in any mental representation or transition. Experts 
showed significantly smaller averages of unproductive actions than students in every mental representation, except 
in CM. In this mental representation, the amount of actions taken by experts was so large that the low proportion 
of unproductive actions in CM (only 6%) implied an average comparable to the students’ averages. 
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Finally, the individual distributions of actions among mental representations were compared within each group 
using the SD/Mean ratios for productive and unproductive actions in each mental representation and transition 
considered in the model. The respective ranges for these ratios in productive actions were 0.33-1.07 for experts, 
0.42-1.73 for advanced students and 0.44-2.41 for initial students. Correlations between pair of participants were 
computed using the 16 values for the productive and unproductive types of actions. The mean value of the cor- 
relations between pairs of advanced students was 0.57 (ranging 0.14-0.93) and a similar mean value was obtained 
for initial students 0.54 (0.07-0.89). However, the mean correlation between pair of experts was 0.90 (0.69-0.98): 
in this case, unproductive actions were not considered due to the frequent 0 value. Hence, the agreement among 
experts seemed to be higher than the agreement among initial students or advanced students. The distributions 
were averaged for experts, and for initial and advanced students, and the 16 averages were then compared us- 
ing the Pearson correlations. Experts’ averages were moderately correlated with the ones of advanced students 
(r(16)= .52), but the correlation was clearly lower with the initial students’ averages (r(76)= .15). Initial and advanced 
students’ averages were highly correlated (r(16)= .75). 


Discussion 


First, the proposed model to analyse experts’ and novices’ problem-solving skills accounted for all the informa- 
tion provided by the participants. The inter-rater reliability in the different classifications reached satisfactory values. 

Second, the model accounted for expert-novice differences and described these differences in terms of 
the content (the solving actions) of the different mental representations elaborated and inter-related by solvers. 
Experts showed a higher global average of solving actions than initial (2"4 year) and advanced (5" year) students. 
Experts took more productive actions than advanced students, and the latter took more productive actions than 
initial students. However, the average of unproductive actions of advanced students was equivalent than the one 
of initial students, and higher than experts’ average. 

In the present research a certain progression with the university year was observed in terms of productive 
actions taken, as expected. Despite this, an expected decrease of unproductive actions did not appear: the differ- 
ences between advanced and initial students almost vanished when the averages per participant were compared. 
In addition, advanced students’ averages showed a medium strength correlation with experts’ averages and a 
medium-high strength correlation with initial students’ averages. Advanced students showed similarities with 
experts mostly in productive actions, and similarities with initial students regarding unproductive actions. Thus, 
university students seemed to learn suitable solving procedures along the years, but the learning was insufficient 
to solve undefined problems with an implicit scientific model. In fact, in the present research significant differences 
in problem-solving actions were found between experts, advanced students, and initial students in all types of 
actions taken to elaborate and develop the conceptual (scientific) model. 

Slight differences in problem solving actions between initial students and graduates were observed in Kohl 
and Finkelstein’s (2008) study on problem-solving in physics. Using a different taxonomy, Rosenzweig et al. (2011) 
also differentiated productive from unproductive metacognitive actions when comparing problem-solving of 
middle-school students with learning disabilities, low-achieving students, and average-achieving students. When 
the problem stated was difficult, the mean number of verbalized productive actions decreased as the group ability 
increased, and the amount of verbalized unproductive actions decreased as the group ability increased. Results in 
the present research show a similar trend. 

According to the types and amounts of actions associated with each mental representation or transition, 
initial students, advanced students and experts showed different behavioural patterns. A summary is provided in 
the following lines. 

Elaboration of a useful Situation Model (SM): Although the experimental problems were selected to avoid 
lack of understanding at SM level, students in general elaborated this representation by dealing with features and 
by activating prior experiences in the real world. Initial and advanced students showed significant percentages 
of orientating and planning actions while developing the SM. Moreover, in initial and advanced students nearly 
20% and 25% of these actions respectively were unproductive. This is coherent with previous findings in the sense 
that novices tend to be disturbed by surface details instead of structural information (Boshuizen et al., 2006; Chi 
et al., 1981; Randles & Overton, 2015). Conversely, experts took fewer actions associated with the SM (3% of the 
total actions). 

Building the Conceptual (scientific) Model (BCM): Students took relative high amount of actions in BCM 
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trying to identify the appropriate scientific model and the relevant magnitudes implied. Most advanced students 
succeeded in this effort, but most initial students did not. Experts built the scientific model more quickly: in a 
hypothesizing process, they activated a previously known conceptual model and then tested their suitability 
by running it. Experts did not encounter obstacles transferring from the situation model to the conceptual 
model. Their quick and efficient coding allowed them to choose the physics or chemistry model efficiently and 
to activate a vast prior knowledge to go ahead. In agreement with previous outcomes (Ball, et al., 2004), experts 
showed a schema-driven behaviour, and the schema was a well-known scientific model. 

Developing the Conceptual Model (CM): Experts elaborated a broader conceptual scientific model in com- 
parison to the one elaborated by advanced or initial students. Most of experts’ solving actions (40%) were taken 
in order to develop this mental representation. As said before, experts tested the hypothetic scientific model 
built before by running it in their minds, considering extreme situations and using approximations instead of 
specific numerical values for the magnitudes. The pre-eminence of CM over other mental representations when 
solving undefined problems was also observed by Truyol et al. (2014) in their analysis of expert physicists. In 
comparison, advanced and initial students devoted lower percentages out of the total actions taken in the CM 
mental representation (advanced students 17%; initial students 13%). The main difference in productive actions 
between experts and students was given in this mental representation. Most initial students were unable to 
elaborate and/or run a suitable scientific model in their minds. Particularly, most of them were unable to identify 
the appropriate magnitudes to describe the situation in the problem, and only a few were able to determine 
the appropriate magnitude for the unknown. Although some advanced students found similar difficulties, they 
were able to move on and elaborate connections among magnitudes. 

Building the Formalized Model (BFM): Most initial students expressed wrong or inappropriate relations 
among magnitudes. They also failed at identifying appropriate data in tables or graphs or interpreted these 
data wrongly. All of them failed to address the absence of quantities and were unable to assign appropriate 
quantities to the relevant magnitudes. Most gave up and quit in this phase. Some advanced students also used 
ineffective equations although none wrote incorrect equations. Advanced students also claimed for specific 
quantities to achieve a solution for the problem. In fact, they were able to assign reasonable numerical values 
to the magnitudes in order to advance with the problem-solving process. Randles and Overton (2015), and 
years before Gil-Perez and Torregrosa (1983), found that novices tend to go quickly on mathematical equations 
when solving chemistry or physics problems respectively. Therefore, when numerical data for the relevant and 
well-defined magnitudes were not provided in the problem statement, novices showed difficulties to advance. 
In the present research all the students, advanced or initial, quit the problems without achieving a correct and 
complete solution in formal terms. 

From this stage on, only experts were able to correctly perform the last solving steps: developing the Formal- 
ized Model (FM), Interpretation (IN) and finally Instantiation (IS) to connect the problem outcomes to real world. 
Novices showed troubles to correctly interpret what they were computing, the graphs or the numerical data 
they were looking for. In fact, the unproductive actions associated with Interpretations doubled the productive 
ones. In addition, novices provided few correct elaborations connecting science to the real world. 

As the analysis of solving actions taken concerns, the results obtained in the present research are comparable 
to the ones reported in Randles and Overton (2015). Although categories in both studies cannot be related in 
a univocal way, quantitative results show that professor/student differences in percentages were of the same 
sign in both studies. Qualitatively, this means that experts overcome students in the same types of actions and 
vice versa, although the respective percentages were different. One disagreement between the present research 
and Randles and Overton's was observed in the category related to ‘dealing with equations and calculations’. In 
Randles and Overton (2015), undergraduates showed a higher mean percentage over professors, but in the 
present research they showed similar mean percentages. This difference might be explained by the nature and 
difficulty of the experimental problems posed to the undergraduates. In the present research, novices were 
unable to develop the formalized model due to a lack of numerical data in the problem statement. Thus, they 
dealt with equations in a lesser amount than experts. 

Additional outcomes support and describe previous findings in terms of a cognitive basis. Experts showed 
higher similarity among their individual behaviours than did initial or advanced students, in agreement with the 
outcomes obtained by Hong and Liu (2003). These authors concluded that experts used a similar schema-like 
piece of knowledge to analyse and solve the posed problem. This might explain the high agreement among 
experts, as compared with students. This schema-driven behaviour has been also exposed by Larkin et al. (1980) 
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in expert physicists. In the present research, the mean agreement among pairs of initial students or among pairs 
of advanced students was similar. This suggests that in academic problem-solving the state of expertise is quite 
homogeneous regarding the solver’s behaviour, while intermediate states show greater levels of heterogeneity 
due to the variability in individual’s knowledge and learning rhythms. 

The students were able to elaborate, use and connect all the mental representations and transitions consid- 
ered, as experts did. However, when the content and quality of the representations were analysed, large differ- 
ences emerged. These results are in agreement with Kohl and Finkelstein’s (2008) conclusions in their study on 
the way experts and novices use multiple representations in physics problem solving. The links between different 
mental representations while solving design problems were the focus in Bjorklund’s (2013) research, and also in 
Kohl and Finkelstein’s (2008) research on physics problem-solving by initial university students and graduates. In 
the present research, explicit connections between representations necessary to solve the problem, or to make 
sense of the outcomes, were coded as: a) EXecuting actions associated to Building the Conceptual Model or to 
Building the Formalized Model; and also as: b) EValuation and ELaboration actions, associated to INterpretations 
and InStantations respectively. Experts’ average and percentage of these explicit productive connections were 
clearly higher (12.2 actions/expert; 19.3% out of total actions taken by experts) than the novices’ corresponding 
values (3.5; 8.4%). Indeed, experts successfully linked different mental representations more frequently than 
novices, which is in line with evidence reported by the previously mentioned studies. 


Conclusions, Limitations and Possible Implications 


The present exploratory research aimed at answering three research questions. The answers to these ques- 
tions implied using the proposed model to describe and compare the experts’ and students’ distributions of 
solving actions in the different mental representations necessary to understand and solve undefined physics 
or chemistry problems, i.e. problems in which the statement does not provide the scientific model or data, but 
only a problematic situation in the real world. 

Naturally, differences between experts and students were evident in the solutions achieved, as well as in the 
distributions of solving actions among mental representations. These distributions pointed out specific students’ 
lack of expert knowledge. The main shortfalls were detected in the development of the mental representation 
Conceptual (scientific) Model, where the expert-student differences were large, even for advanced students. 
This suggests that university physics and chemistry curricula should help students develop the important skill 
of modelling reality with science in a more effective way. In addition, advanced students’ distributions were 
similar to experts’ distributions in some productive actions, but they were very similar to initial students’ distri- 
butions in unproductive actions. However, differences between advanced and initial students were observed 
in productive actions associated with the Conceptual and Formalized Models. This implies that throughout 
the university degrees students acquire new important skills, but it also draws attention to the effectiveness 
of university curricula to correct students’ erroneous conceptions present at the initial courses and to provide 
them with applied knowledge. 

In addition, experts’ distribution of actions in the mental representations showed high level of agreement, 
whereas the respective levels of agreement among initial or advanced students were clearly lower. This suggests 
that experts share a similar knowledge and that it is used in a schema-driven way and, therefore, expertise is 
characterized by knowledge-in-action schemata. 

As said before, these provisional results lack enough external validity and a larger volume of data from more 
representative samples are needed before a strong suggestion can be made to teachers regarding what needs 
to be done. The experimental problems should be more varied as well. Notwithstanding, the present research 
showed convergence with similar previous studies in their qualitative results, which encourages future research 
in this direction. If the obtained differences between experts, advanced and initial students were replicated, 
university teachers could improve students’ education by focusing on these outcomes. 
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Appendix. Examples of problems and selected fragments of the interviews 


Physics problem: Two identical racing cars arrive at the same time to the last corner of the circuit, after which is 
the finish line. Both cars describe a perfect semi-circular curve on the track without sliding, one moving “inside” with a 
smaller radius, and the other “outside” with a larger radius. Please reason which car will win. 




















Units (Expert’s protocol. Units are not necessarily in order) Action Say Utility 
(...) One of the cars is moving “inside” and, the other one, it moves “outside” in parallel trajectories... OR SM PRO 
The described curves are perfect semi-circles of 180 degrees for both cars...Then, this is a case of OR BCM PRO 
circular motion. 
The centripetal force equals the mass times the speed squared, divided by the radius... (writes Fc= m1 EX BEM PRO 
vi2/R1). 
Here, v1 is the speed of car 1 when it is describing the curve without sliding. MO CM PRO 
v1 will be the higher when the friction force is the maximum possible for the car without sliding. EX CM PRO 
From here | will obtain the speed of the car 1, v1. PL FM PRO 
(...) As the radius R1 is greater than R2, then the speed v1 is greater than v2 EV IN PRO 
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: ; : nn ‘ Mental a 
Units (Student’s protocol. Units are not necessarily in order) Action Repres Utility 
Here, the statement says that both cars arrive at the same time (he/she underlines the sentence in the 
PL MS PRO 
statement), ... 
... this means that speed 1 is equal to speed 2 (writes v1 = v2) EX CMC UNP 
Here it says that the cars handle the curve without sliding (he/she underlines the sentence in the state- 
PL MS PRO 
ment), ... 
... does this mean that there are no other forces acting on the cars? EX CMC PRO 
As the distance is larger for one car than for the other one, the car moving “inside” will arrive first. EL IS UNP 
Interviewer: Can you suggest a solution from the information provided in the problem? (Thinks for a few MO CMF UNP 


seconds) No, because | do not remember the equations for the circular motion. 


Chemistry problem: A clean steel wok is sprinkled with some drops of water. Some time later, a reddish-yellowish 
substance appears under a water drop. Using your knowledge in chemistry, please explain the observed change. 





Units (Expert’s protocol. Units are not necessarily in order) Action ee Utility 
Probably it is a redox reaction. | mean, the iron has to be oxidized because its colour has changed. OR BCM PRO 
Obviously for the iron to oxidize there must have been a reduction process in the other way. (Writes EX cM PRO 


while speaking Fe: oxidation; H2O: reduction). 





(He/she schematically draws the reduction process on the paper). The electrons that go from one side 
to the other obviously came out of the iron, and there is some species here (he/she points to the water EX BFM PRO 
half-reaction he/she just wrote) that must have taken the electrons. 





(He/she writes the equality of concentrations between protons and oxydryl: [OH-]= [H+] while speaking) 























The concentration of oxydryls is equal to the concentration of protons... S EM FRG 
This diagram (a Pourbaix diagram deliberately searched for and found in internet) indicates that if | go EV IN PRO 
up this line (points to the y-axis) what is there (the wok’s oxide) must be more or less in this region. 
If this (he/she refers to the reddish-yellowish substance in the wok) is soluble, probably we have a 

ee EL IS PRO 
Fe203 oxide with n water molecules. 

; : ‘ aan : Mental a 
Units (Student’s protocol. Units are not necessarily in order) Action Repres Utility 

Can | look up [the information] in books? OR NM PRO 
Let me think, because this thing that looks black (he/she points the image), isn’t it the soot that gets OR SM UNP 
formed in the pot when it reacts with heat? The pot goes bad. 
Can't it be like an oxidation or something that the water produces on the pot’s steel? PL BCM PRO 
I'm going to look up (in the book) the value of the oxidation potential of water ... to find out whether it is PL BEM PRO 
an oxidant or not. 
I'm sorry, but | haven’t started studying physical chemistry yet (...) | feel that I’m lost; I'm not going MO NM UNP 
anywhere. 
| know there is water: (He/she writes 2H20 + 2e- +H2 + 20H-) EX FM PRO 
(He/she looks at the reduction potentials table) The potential for iron is negative. PL BFM PRO 
Let's write it down. (He/she writes: Fe — Fe2+ + 2e-; E°= -0.44V) EX FM PRO 
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SEY EEE TNA AE UD Ay ONO CORR R ENR RCUND RNR ISON IGAS 498, sey 
(pp. 730-746) ISSN 2538-7138 soniines 
Units (Student’s protocol. Units are not necessarily in order) Action coe Utility 
If the potential is negative, it means that the reaction does not take place, or that for taking place it 
: : ae : EV IN UNP 
needs energy, or temperature. Because | believe that it happens anyway, but it is not favourable (sic). 
But if the delta of the cell redox potential is a positive value, it is true that every time you put a drop of 
are EL IS UNP 
water, the pot stains like this. 
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